The present invention is directed to distributed, multi-node data processing systems. More particularly, the present invention is directed to a mechanism for managing a plurality of diverse resources whose presence on remote external data processing nodes can lead to situations in which their status is either changed, unknown or not well defined. Even more particularly, the present invention is directed to a scalable resource management method for resources on remote nodes in a manner which is tolerant of management node failures and which is especially useful in systems having a very large number of nodes.
The present invention is an improvement upon a previously submitted patent application (Ser. No. 10/227,254, filed Aug. 23, 2002) titled “A Monitoring Method of the Remotely Accessible Resources to Provide the Persistent and Consistent Resource States,” which provides a method for continuously monitoring and controlling operations in a distributed, multi-node data processing system against the possibility of a failure at one of the monitoring nodes. The present invention is an enhancement of the methods described previously in that it provides protection against failure by providing a failover mechanism for the resource manager's functionality to other nodes in the system.
In distributed systems, many physical and logical entities are located throughout the nodes in the system, including nodes that are best characterized as being remote nodes that may or not be closely coupled to other nodes in the system. This is one of the typical characteristics found in distributed systems. In the present context, these physical and logical entities are called resources in the present context. The term “resource” is used broadly in the present description to refer to software as well as to hardware entities. Examples of such resources include ethernet device eth0 on node 14, the database table “Customers,” IP address 9.117.7.21, etc. From these few examples it is thus seen that the term “resources” is intended to refer to a number of highly, disparate entities found in distributed, multi-node data processing systems. Each resource, however, has at least one attribute which at least partially defines a characteristic of the resource. Moreover, some of the attributes are reflected through the status or condition of the resources. As an example, a network ethernet device typically includes attributes like name (for example, eth0), OpState (for example, Up, Down, Failed, etc.), its address (for example, 9.117.7.21), etc. Thus name, OpState, and address are referred to as resource attributes. The resource attributes which reflect the status of the resources are categorized as dynamic attributes (like OpState above).
Because these resources provide services to other components of the distributed data processing system (for example, to system management tools or to user applications, etc.), these resources need to be monitored and/or controlled at the system level by an entity that is typically referred to, in the present description, as the “Resource Manager Infrastructure” or “RMI.” Most RMIs assume that the resources as described earlier and that the RMIs are contained or confined to the same node. However, the current assumptions of the same node resources and RMIs do not hold if certain systems have different types of nodes which may or may not have both the resources and RMIs because of software, hardware and/or architectural limitations. As used herein, the term Resource Manager Infrastructure refers to a system of software components comprising subsystems which perform monitoring, querying, modifying, and controlling functions for resources within a cluster of data processing nodes. The subsystems typically include shared components that are used by cluster resource managers for handling common activities like communication between nodes, registration of the resources, etc. RMIs also include resource managers (RMs) which provide methods for operating on a resource; these operations include monitoring the state of the resource and controlling the resource configuration and the behavior of the resource. Typically any given RM deals with a set of similar types of resources.
The previously submitted application, referred to above, provides a mechanism to monitor and control the remotely accessible resources which exist on the non-RMI nodes through the concept of proxy resource manager and proxy resource agent, and also provides the persistent and consistent dynamic attributes even after the proxy resource manager fails and is restarted. However, in the case of a (relatively) permanent failure of the proxy resource manager or its node, the remote resources cannot be continuously monitored or controlled, even if the remote resources are still up and active.
The present invention provides continuous services for monitoring and controlling those remote resources even in the event of a failure of the proxy node (which runs the proxy resource managers) in addition to providing consistent and persistent resource attributes.
Furthermore, if the size of the distributed data processing cluster which is desirably subject to resource management is large (for example, one-thousand nodes or more), a failover system should also be adequately scalable, particularly when messages are to be exchanged amongst all of the nodes in a large system. The present invention proposes an efficient mechanism which provides failover resource management functionality to support large clusters of nodes.
Use of the present invention provides the following representative advantages:                (1) The proxy resource manager (PxRM) is transparently moved to other nodes so that the remote resources are continuously monitored and controlled without disruption.        (2) When the failed node is repaired, the proxy resource manager is moveable back to the originally failed node.        (3) The method is efficient and scalable in terms of the load balancing, the recovery speed, and the number of nodes.        (4) Therefore, in combination with the previously mentioned inventive method provided in the above-mentioned application for “A Monitoring Method of the Remotely Accessible Resources to Provide the Persistent and Consistent Resource States,” remote resources herein are persistently monitored so as to maintain consistent attribute values, against any failures, which includes the proxy resource manager, or Resource Management Infrastructure (RMI) subsystem, and/or other nodes, in a scalable and efficient manner.        